The following patents and publications, the subject matter of each is being incorporated herein by reference in its entirety, are mentioned:
U.S. Pat. No. 6,999,600, issued Feb. 14, 2006, by Venetianer et al., entitled “Video Scene Background Maintenance Using Change Detection and Classification,”
U.S. Pat. No. 6,625,310, issued Jan. 17, 2006, by Lipton et al., entitled “Video Segmentation Using Statistical Pixel Modeling,”
U.S. Pat. No. 6,696,945, issued Feb. 24, 2004, by Venetianer et al., entitled “Video Tripwire,”
U.S. Published Patent Application No. 20060268111, filed May 31, 2005, by Zhang et al., entitled “Multi-State Target Tracking,”
U.S. Published Patent Application No. 20070127774, filed Jun. 7, 2007, by Zhang et al., entitled “Target Detection and Tracking from Video Stream,”
U.S. Published Patent Application No. 20050146605, filed Nov. 15, 2001, by Lipton et al., entitled “Surveillance System Employing Video Primitives,”
U.S. Pat. No. 6,064,827, issued May 16, 2000, by Yasuhiro Toyoda, entitled “Image Stabilizer,”
U.S. Patent Application No. 20050179784, filed Aug. 18, 2005, by Yingyong Qi, entitled “Adaptive Image Stabilization,”
R. Collins, A. Lipton, H. Fujiyoshi, and T. Kanade, “Algorithms for Cooperative Multisensor Surveillance,” Proceedings of the IEEE, Vol. 89, No. 10, October, 2001, pp. 1456-1477,
Jesse S. Jin, Zhigang Zhu, Guangyou Xu, “A Stable Vision System for Moving Vehicles”, IEEE Transactions on Intelligent Transportation Systems, vol. 1, no. 1, March 2000, pp 32-39.
Video content analysis (VCA) may apply computer vision and artificial intelligence algorithms to video streams. Various applications for VCA include, for example, data retrieval and intelligent video surveillance (IVS). Recently, video surveillance have become more critical in many areas of life. One problem with video as a surveillance tool is that the video may be manually intensive to monitor. VCA algorithms may be applied to automate the video monitoring in the form of intelligent video surveillance systems. Such solutions are described, for example, in U.S. Pat. No. 6,696,945, U.S. Published Patent Application No. 20050146605, or U.S. Published Patent Application No. 20060268111, identified above.
One component in an IVS system may be referred to as background modeling, which may be used to differentiate between foreground and background, detect changes in the scene, and detect targets of interest. Pixel-based background modeling may be used in current video surveillance systems such as described, for example, in U.S. Pat. No. 6,999,600, U.S. Published U.S. Pat. No. 6,625,310, and R. Collins, A. Lipton, H. Fujiyoshi, and T. Kanade, “Algorithms for Cooperative Multisensor Surveillance,” identified above. Pixel-based background modeling may use an individual image pixel as the base unit to model to the background scene. Pixel-based background modeling may be considered to be a widely used approach and may work well in many scenarios. However, pixel-based background modeling is far from perfection, especially compared with human perception in some less-friendly environments.
As one example, in a video content analysis system analyzing video from a static camera, a change detection module of the video content analysis system may employ pixel-based background modeling. Some camera-related video phenomena may significantly increase the difficulty of the change detection module to detect change in the video and may, in fact, reduce the overall system performance. Camera automatic gain control (AGC) and camera jitter, for example, may be two of the most common causes of these difficulties.
As to the first possibility, difficulties with change detection may arise from camera AGC. For instance, in video surveillance security applications, many surveillance cameras are fixed static cameras. To perform automatic video content analysis for these scenarios, the video content analysis system may usually assume that the background of the video is stable so that any fast changes in the video frames may be assumed to indicate moving targets, which are often the objects of interest. However, if anything triggers camera AGC, the video frames may include significant global intensity changes, including changes in the spatially stable background area. For example, in many video surveillance situations, the camera AGC may be triggered by large size moving targets appearing in the camera view whose image intensity is either much higher or much lower than the corresponding background area. Without detecting and accurately compensating these AGC effects, a VCA-based IVS system may likely introduce significant false target detections, where some background regions may be considered as foreground objects due to the fast intensity change caused by the camera AGC mechanism. One difficulty in such a system may be that given an image pixel whose intensity value changed from frame to frame, the system may need to determine whether this intensity change is caused by camera AGC or whether the pixel is part of a foreground moving object. However, current VCA-based IVS systems typically do not compensate for or consider this possibility.
As to the second possibility, difficulties with change detection may arise from camera jitter. Current image stabilization methods such as described, for example, in U.S. Pat. No. 6,064,827, U.S. Patent Application No. 20050179784, and Jesse S. Jin, Zhigang Zhu, Guangyou Xu, “A Stable Vision System for Moving Vehicles,” identified above, may work on aligning consecutive video frames from a non-stationary camera to provide a steady view in an attempt to overcome camera jitter. In some applications, the camera jitter may be significant, for example, with handheld video cameras, but the requirement for the frame alignment accuracy may not be so critical. In other applications, however, the camera jitter may be significant and caused by wind or platform vibrations, and the requirement for the frame alignment accuracy may be critical. In these applications, the current stabilization techniques may fail when camera jitter occurs and when a large moving foreground object is in the camera view.